feat: Support for fault-tolerant execution by dervoeti · Pull Request #779 · stackabletech/trino-operator

dervoeti · 2025-08-04T12:28:02Z

Description

Implementation for #576

When providing credentials for S3, Azure or GCS we currently need a SecretClass. It would be easier if the user could just provide a Secret as an alternative, we had that discussion recently. To be consistent with the current state, I used SecretClass exclusively for now.

Things I did not do in this PR to ease the review process (we could tackle those as a separate PR):

Combine redundant logic from FTE and Catalog modules
Use config-utils instead of doing the load_env_from_files stunt

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

deploy/helm/trino-operator/crds/crds.yaml

sbernauer

I dropped some comments on the CRD code and YAML.
As an possible alternative I can propose a complex enum style, where we can together think about it, just pasting it here for reference for now

spec:
  clusterConfig:
    faultTolerantExecution:
      query:
        retryAttempts: 4 # maps to query-retry-attempts
        retryInitialDelay: 10s
        retryMaxDelay: 60s
        retryDelayScaleFactor: 2.0 # f32
        exchangeManager: # Optional
          deduplicationBufferSize: 64Mi # Quantity
          encryptionEnabled: true
          sinkBufferPoolMinSize: 20
          sinkBuffersPerPartition: 4
          sinkMaxFileSize: 2Gi # Quantity
          sourceConcurrentReaders: 8
          s3:
            baseDirectories:
              - s3://trino-exchange-bucket/spooling
            connection: # Mandatory
              reference: minio-connection
            maxErrorRetries: 10
            uploadPartSize: 10Mi # Quantity
      # OR
      task:
        retryAttemptsPerTask: 4 # maps to task-retry-attempts-per-task
        retryInitialDelay: 10s
        retryMaxDelay: 60s
        retryDelayScaleFactor: 2.0 # f32
        exchangeManager: # Mandatory
          # ... same struct as above

docs/modules/trino/pages/usage-guide/fault-tolerant-execution.adoc

tests/templates/kuttl/fault-tolerant-execution/helm-bitnami-minio-values.yaml

examples/simple-trino-cluster-fault-tolerant-execution.yaml

rust/operator-binary/src/crd/fault_tolerant_execution.rs

…adoc Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>

…nge manager

docs/modules/trino/pages/usage-guide/fault-tolerant-execution.adoc

tests/templates/kuttl/fault-tolerant-execution/04_check-fte.py

tests/templates/kuttl/fault-tolerant-execution/05-run-tests.yaml

tests/templates/kuttl/fault-tolerant-execution/02-install-trino.yaml.j2

…adoc Co-authored-by: Malte Sander <malte.sander.it@gmail.com>

maltesander

LGTM!

dervoeti · 2025-08-21T15:30:05Z

Release note

SDP now supports configuring fault-tolerant execution for Trino via the TrinoCluster CRD

lfrancke · 2025-09-08T09:52:29Z

Are there any docs for this we can link to?

sbernauer · 2025-09-08T10:12:05Z

Yes, https://docs.stackable.tech/home/nightly/trino/usage-guide/fault-tolerant-execution/

sbernauer reviewed Aug 5, 2025

View reviewed changes

deploy/helm/trino-operator/crds/crds.yaml Outdated Show resolved Hide resolved

dervoeti force-pushed the feat/fault-tolerant-execution branch 2 times, most recently from 4cc640f to 3d113df Compare August 5, 2025 18:44

dervoeti self-assigned this Aug 5, 2025

dervoeti moved this to Development: In Review in Stackable Engineering Aug 5, 2025

dervoeti added this to Stackable Engineering Aug 5, 2025

dervoeti force-pushed the feat/fault-tolerant-execution branch from fc684e3 to 85a09cc Compare August 5, 2025 19:40

sbernauer reviewed Aug 6, 2025

View reviewed changes

dervoeti force-pushed the feat/fault-tolerant-execution branch 2 times, most recently from 2567920 to 843b18d Compare August 7, 2025 14:06

sbernauer reviewed Aug 12, 2025

View reviewed changes

rust/operator-binary/src/crd/fault_tolerant_execution.rs Show resolved Hide resolved

dervoeti force-pushed the feat/fault-tolerant-execution branch from fd42c2b to af39e6b Compare August 12, 2025 12:28

lfrancke linked an issue Aug 13, 2025 that may be closed by this pull request

Support Fault tolerant execution (FTE) #576

Closed

3 tasks

dervoeti force-pushed the feat/fault-tolerant-execution branch from eaaff12 to 9e587d8 Compare August 13, 2025 11:39

dervoeti requested a review from sbernauer August 13, 2025 11:44

sbernauer moved this from Development: In Review to Development: Waiting for Review in Stackable Engineering Aug 18, 2025

sbernauer requested review from maltesander and removed request for sbernauer August 18, 2025 07:07

sbernauer moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Aug 18, 2025

dervoeti and others added 11 commits August 18, 2025 10:20

CRD update

dcb6f32

feat: fault tolerant execution

d3f9322

test: fault-tolerant execution integration test

96ebd55

docs: fault-tolerant execution documentation

04d31cb

chore: changelog

5a247ff

fix: lint fixes

39eecfa

Update docs/modules/trino/pages/usage-guide/fault-tolerant-execution.…

ee145e0

…adoc Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>

fix: fixed review feedback

47d2ef5

feat!: remove explicit Azure and GCS support

056f84b

feat: use PascalCase for Query/Task / allow configOverrides for excha…

aefba45

…nge manager

fix: always convert durations to seconds

499665c

dervoeti added 4 commits August 18, 2025 10:20

fix: moved to quantities in the FTE docs example

97d2d1f

fix: moved to quantities in the FTE docs

fab0e45

chore: cargo fmt

096afe6

chore: pre-commit fix

b9f5ca9

dervoeti force-pushed the feat/fault-tolerant-execution branch from 539473f to b9f5ca9 Compare August 18, 2025 08:20

maltesander requested changes Aug 18, 2025

View reviewed changes

dervoeti and others added 6 commits August 18, 2025 14:20

Update docs/modules/trino/pages/usage-guide/fault-tolerant-execution.…

8afac31

…adoc Co-authored-by: Malte Sander <malte.sander.it@gmail.com>

Update docs/modules/trino/pages/usage-guide/fault-tolerant-execution.…

30baa69

…adoc Co-authored-by: Malte Sander <malte.sander.it@gmail.com>

Update docs/modules/trino/pages/usage-guide/fault-tolerant-execution.…

0d9e066

…adoc Co-authored-by: Malte Sander <malte.sander.it@gmail.com>

Update docs/modules/trino/pages/usage-guide/fault-tolerant-execution.…

684083c

…adoc Co-authored-by: Malte Sander <malte.sander.it@gmail.com>

fix: integration test fixes

cef8eac

fix: integration test fixes

f4a438a

maltesander approved these changes Aug 18, 2025

View reviewed changes

dervoeti added this pull request to the merge queue Aug 18, 2025

dervoeti added demo-needed scheduled-for/25.11.0 labels Aug 18, 2025

Merged via the queue into main with commit 28260dc Aug 18, 2025
16 of 17 checks passed

dervoeti deleted the feat/fault-tolerant-execution branch August 18, 2025 14:35

dervoeti mentioned this pull request Aug 19, 2025

fix: mention exchange manager overrides #790

Merged

20 tasks

dervoeti moved this from Development: In Review to Development: Done in Stackable Engineering Aug 19, 2025

dervoeti removed the demo-needed label Aug 26, 2025

lfrancke added release-note Denotes a PR that will be considered when it comes time to generate release notes. release/25.11.0 and removed scheduled-for/25.11.0 labels Sep 8, 2025

lfrancke moved this from Development: Done to Acceptance: In Progress in Stackable Engineering Sep 8, 2025

lfrancke moved this from Acceptance: In Progress to Done in Stackable Engineering Sep 8, 2025

Uh oh!

Conversation

dervoeti commented Aug 4, 2025 • edited by maltesander Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

Uh oh!

sbernauer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dervoeti commented Aug 21, 2025

Release note

Uh oh!

lfrancke commented Sep 8, 2025

Uh oh!

sbernauer commented Sep 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dervoeti commented Aug 4, 2025 •

edited by maltesander

Loading